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ABSTRACT 

This paper discusses the Research and Evaluation Program at 
the State Library of New South Wales (Australia) and what it has revealed 
about clients' notions of value and relevance. An overview is provided of the 
services that have been developed in order to meet clients expectations, 
including r^eading room services and the development of the 3Tier service 
model comprising: (1) simplifies the process of getting started, providing 

tools such as guides that allow independent access and services clients can 
access for themselves; (2) offers assistance in getting clients started on 
beginning level research through offering client education program^, 
conducting straightforward research on their behalf, or providing 
professional advice on using the collection of the library; and (3) offers 
formalized referral to specialist sections, subject experts, or curators and 
tailors and customizes services to readers* needs . The library* s approach to 
benchmarking library services is outlined, including a model that shows the 
relationship between the variables of client satisfaction, the X factor 
(i.e., client's perception of the library*s value), price, service, and 
product. A chart presents key service attributes and the top ten factors for 
quality service. (Contains 12 references.) (MES) 
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Segment argument 

The focus of PMs should demonstrate library value and relevance and be the basis for comparison with 
competitors in other industries. 

Our approach 

Performance measurement and benchmarking are two activities that are part of a broader review of services 
that the State Library of NSW has embarked upon. This review is based on two key components: the 
Library's Research and Evaluation Program and Quality Initiatives Program. The review is, in part, a 
reflection of the unprecedented changes in the information environment, as well as increasing client 
expectations and the growth in remote client use. These changes require a responsiveness that questions 
definitions of the Library's value and relevance within the current and future information environment. 

In our presentation today we would like to discuss some of the ideas and issues that have been raised in our 
review regarding concepts such as value and relevance in a library context. In particular, our central 
premise is that while these concepts are two important measures, the critical issue is ascertaining how 
notions of value and relevance are defined and by whom. At the State Library of NSW we define these 
concepts from a client perspective. We consider it is imperative that we are aware of the range of 
perceptions and ideas associated with such concepts, including social and emotive connotations. As these 
perceptions, often unarticulated, have a major impact on clients' overall perceptions of the Library and the 
services we offer. 

With regard to benchmarking, when considering competitors in other industries, our experience suggests 
that specific service attributes are a more appropriate basis for comparison than broader concepts such as 
value and relevance. In this presentation we will outline how we define, measure and benchmark these 
attributes. 

Our presentation will commence with a discussion of the Library's Research and Evaluation Program and 
what it has revealed about clients' notions of value and relevance. We will then provide a brief overview of 
some of the services that have been developed in order to meet client expectations, before outlining our 
approach to benchmarking Library services. 

Client research and evaluation 

The overall aim of the Library's Research and Evaluation Program is to develop comprehensive information 
on Library clients - who they are, why and how they use the Library and what they think of current and 
possible future services. Projects conducted have employed a variety of methods to gather feedback from 
clients, including in-depth interviews, focus groups and questionnaires. 

The projects have highlighted the diverse client base that uses the Library and why it is important to 
segment clients when considering performance measurement. At the State Library of NSW our clients range 
from the novice, first time user to the experienced researcher and scholar. And while caution must be 
exercised in categorising our clients too broadly, they can be segmented into three different, but not 
mutually exclusive categories: the reading room client, the recreational client and the remote client. This 
paper will focus on the reading room client. 

The reading room client includes both Mitchell Library readers and State Reference Library (SRL) users. 
While these clients may view an exhibition, attend a Library event, or use our remote services, their primary 
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reason for using the Library is to conduct research in either of the two reading rooms. 

In conducting research with our reading room clients, we have discovered the complex, diverse and often 
contradictory meanings underpinning their notions of the Library's value and relevance. We will now 
highlight some of these perceptions, before turning to how we have attempted to address and measure 
them. 

Mitchell Library 

Our research suggests that amongst Australian libraries, the Mitchell Library accords 'very' valuable and 
relevant ratings. This is due to its origins, history and resources. For scholars, particularly in Australian and 
south - west pacific history, the Mitchell Library is recognised internationally as one of the most extensive 
sources of original material. 

In gathering comprehensive information on Mitchell Library readers for the first time, we have been able to 
gain a more insightful understanding of who actually uses the Library. In particular, the Mitchell Library 
provides services to four major reader groups: professional researchers, personal interest readers, students 
and work/business readers. Each group represents a complex mix of perceptions regarding the value and 
relevance of the Mitchell Library. 

The'Mitchell Library's value, according to most professional researchers is defined by a certain culture 
steeped in the notion of the Library as a place of orderly, scholarly learning and research. Assumptions as to 
what constitutes a library, reader and valid (serious) research underpin this culture. Inextricably linked to 
these perceptions is the notion of the Mitchell Library as a constant within a world of change. This projection 
onto the Library as a steadfast institution representing an unchanging culture and values is, in part, a 
reflection of significant change in research and academic circles. In particular, many professional 
researchers feel that the value placed on scholarly research has declined in recent times. The Library, 
according to these readers is valued and relevant due to its role in providing and maintaining a place and for 
significant scholarly research. Change in the Mitchell Library is perceived to represent a decline in the value 
of professional researchers' work, as one reader has remarked: 

"There's a concern amongst the historical profession - in these anti-intellectual times - concern 
that Mitchell Library will lose sight of what it is for". 

However, as the research has revealed, not all readei s share the views held by professional researchers, 
personal interest readers are a case In point. Personal interest readers are usually amateur researchers who 
do not have extensive experience in using the Library (most have been using the Library for less than six 
years). The majority of these readers find themselves at the Mitchell Library, due to their interest in family 
history. In general, personal interest readers are highly motivated and very (narrowly) focused on their 
research. However, unlike professional researchers, these readers define value and relevance in fairly 
pragmatic terms; the delivery of enhanced services for their specific needs. 

Work/business readers share personal interest readers' pragmatic focus. However, there are some 
additional requirements for this group: the delivery of timely services and increased access to electronic 
resources via the Library's website. 

On the other hand, students represent a shift in research 'values'. The way research is conducted and the 
approach taken has changed. Many students have departed from traditional subjects such as history and 
moved to multidiscliplnary fields such as cultural studies. Students studying such subjects approach their 
research from a different, 'postmodernist' framework. The recognition of subjectivity and critical-ficto writing 
is of more value and relevance to these students than 'traditional' notions of objectivity and academic prose. 
This shift affects the way students approach their material. In particular, and much to the chagrin of some 
of their lecturers, not as much emphasis is placed on the use and interpretation of original materials. As a 
result, students tend to draw from more than one source and therefore do not usually place as much value 
on the Mitchell Library as their primary place of research. 

This brief sketch of reader groups reveals some of the challenges the Library needs to meet iri order to 
deliver valued and relevant services to its different reader groups. However, understanding Mitchell Library 
readers' perceptions and the ensuing complexities provides an understanding of only half of the picture. As 
we will now explain, SRL clients hold different Ideas regarding the Library's value and relevance. 
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State Reference Library clients 

Apart from students, it is difficuit to segment the State Reference Library dient. However, while SRL clients 
avoid neat definitions, they are, in most respects unified in their overall perceptions of the Library's value 
and relevance. Unlike many Mitchell Library readers, whose sole focus when discussing the State Library is 
the Mitchell Library, SRL clients tend to take a wider approach and apply their notions of value and 
relevance to the Library as a whole, rather than just the SRL. 

When focusing specifically on the SRL, clients indicate they place the most value on 'efficient' service. 
Efficient service for SRL clients is defined as being able to use the computer catalogue and retrieve the 
appropriate items within a certain amount of time. For students, it means being able to photocopy as many 
references as possible within a set time! 

While efficient services are important to SRL clients, they are just as concerned with br9ader issues 
associated with the Library's value and relevance. In particular, these clients place significant value on the 
Library 'experience'. This 'experience' can mean many different things to different clients however, usual 
definitions are associated with a sense of 'quietness', the pleasure of research and the feeling that they are 
working within a valued 'community' institution. 

Similar to Mitchell Library readers, SRL clients are concerned about change within the Library, albeit from a 
slightly different perspective. Change for SRL clients is defined by technology and the feeling that new 
electronic and virtual technology will lead to a loss of control in accessing and organising information. In 
brief. If we take away the 'physicality' of the Library, in particular, the feeling of touching a book, how does 
the Library experience remain valued and relevant? As some SRL clients have remarked: 

"The future is the net increasing, but you cannot replicate the feeling of touching". 

"I need authenticity, this is iost in the virtual world". 

Future concerns aside, SRL clients also need recognition that a place that they 'value' is valued and relevant 
to the community at large. For while SRL clients are anxious about change within the Library, at the same 
time, they believe the Library needs to promote and accommodate change in the way that ensures it is seen 
as a contemporary and relevant institution amongst other cultural institutions and society in general. 

As the above research findings highlight, in one reading room alone, the concepts of value and relevance 
can mean many different things. More importantly, it is not the differences so much that matter, but the 
tensions that arise in trying to accommodate the various meanings. We will now discuss how we have 
attempted to incorporate clients' perceptions and expectations within the delivery of reading room services. 

Reading room services - the context 

A recent article described libraries as offering a "venue that appeals and nurtures every Idiosyncrasy" and 
that staff within these libraries "are apostles of culture"^. This statement is particularly appealing, 
considering the diversity of the State Library's clients and their requests. But what does it tell us about our 
value and relevance? How can we measure the way we "nurture every idiosyncrasy" or compare it against 
the offerings of our competitors? 

In the past, the strength of a library such as the State Library was based on its Collections and staff who 
knew what was in our Collections. Much more is expected of us now. We are still important as a physical 
entity - a place to visit and see exhibitions or to use the Library's unique collections. However, increasingly 
the shift is away from collections to services that facilitate access and the delivery of information in 
whatever form directly to the client, wherever they are. Technology has enabled services to develop in ways 
that increase client expectations by offering improved choice and convenience. Such changes requirea 
responsiveness that challenges traditional models of reference and research. Our services need to reflect 
these changes. 

This part of our paper focuses on service initiatives, that are a direct result of the client research and the 
quality initiatives project. 

The State Library of New South Wales offers information services from two Reading Rooms; the Mitchell 
Library which holds the Australiana Research Collection and the State Reference Library which offers both 
general reference and specialist services such as the Legal Information Access Centre and the Health 
Information Service. Both Reading Rooms combine to offer information services to remote clients. Recently 
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these two Reading Rooms came under one management structure providing a framework for two different 
libraries to .deliver services in a similar way. 

This single organisational unit is a result of staff from both Reading Rooms participating in a Quality 
Initiatives Project aimed specifically at providing more relevant and effective services to clients. This was 
achieved by reviewing internal processes and work priorities, analysing gaps in our performance and then 
developing better practices and performance in our delivery of services. This information coupled with the 
insights offered by our Research and Evaluation Program has helped us to identify what we have yet to 
accomplish and how we hope to get there. 

As outlined, client research offers us insights into how our clients view what we do. It also helps us 
understand how our clients seek information, rather than how we think they seek Information. This process 
has prompted us to question our service delivery mechanisms. The traditional ways of providing reference 
services that have stood us in good stead over the years are now sometimes inefficient and frustrating for 
both clients and library staff. 

Redefining service 

Until recently, the Library followed a traditional onsite reference model; one that was anchored at physical 
service points. Clients queued for both complex and simple directional inquiries at the same point. Such a 
service symbolised the values of equity and ease of access but it was a one size fits all model. 

Another disadvantage was that continuing a consultation process at the desk with the client was very 
difficult particularly if there were other people queuing behind. Continuity with the same staff member was 
also difficult because of rostering patterns. 

In addition, we recognised that staff were frustrated with this one size model. There were different ways of 
staffing services and different cultures in the two Reading Rooms - one reading room had a team based 
service approach, the other was individualised with staff working independently and with very fixed 
professional and paraprofessional roles. Many of these staff had worked for many years in the same area 
and had built up strong knowledge of the collections. Regular clients quickly "sussed out" who to approach 
for ongoing personal assistance, and would wait until their shift. The paradox in this service was that there 
was a perception of personal service but in reality staff and client were both following a well worn path of 
compatibility that continued existing beliefs and values. 

Our redefinition of services is best described as an iterative, rather than linear process. This approach 
became apparent to us in planning the "new model" process, which involved many discussions with staff and 
with our clients, developing their ideas, talking to other libraries, analysing research results and, most 
importantly, understanding the service from a client perspective though focus group "conversations" with 
our clients. 

These conversations told us that our clients 

• wanted improved convenience and access to our services and collections 

• wanted to become more effective and self sufficient in using our services 

• expected added value in relation to services - value in saving time or evaluative guidance in using our 
electronic products or customised services 

The results of these deliberations was the development of a new service model - what we called a 3Tier 
service model, which defines the levels of service that clients can expect from us and the skills and 
knowledge staff need to provide within these levels of service. 

The 3Tier Service Model 

The Model provides a framework for service delivery in which we offer a variety of service configurations and 
differentiated levels of service that meet the broad range of needs and are supported by skilled staff and a 
more effective use of resources. 

Broadly, the 3Tiers comprise: 

• 1st Tier - aims to simplifying the process of getting started, providing tools such as guides that allow 
independent access, and services clients can access for themselves. 
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• 2nd Tier - offers assistance in getting our clients started on beginning level research through offering 
, dienteducation programs, conducting straightforward research on their behalf, or providing 
professional advice on using the collections of the State Library. 

e 3rd Tier - includes offering formalised referral to specialist sections or subject experts or curators and 
tailoring and customising services to readers' needs. This tier is particularly relevant to people who are 
undertaking long term research in the State Library or who have complex research needs, such as the 
professional researcher. 

Reshaping services is not just about changing the way we do things. It must change the outcome for clients. 
The 3tier model of service only works if it is in a context of value for clients. 

For example, what might constitute value for professional researchers? Elements in all 3 tiers can add value. 
For many, the Library is their workplace so physical spaces, low noise and facilities (space to write, power 
for their laptops) are very important to this group. This group is usually narrow in their research focus and 
are often unaware of new services or products, so we provide them with targeted information through a 
fortnightly newsletter prominently displayed close to where they work. At the other c.nd of the Tier 
spectrum, offering a consultation with a collection spjecialist provides a structured arrangement and a 
guarantee of service to both the researcher and the specialist. 

A tiered framework helps us to plan delivery of services to particular groups of clients (e.g. business people, 
historians, students) in a structured way. Family History clients, like other researchers, use the Library 
primarily for the strength of resources. Staff understand the complexity of products and resources and can 
recommend the appropriate search strategy to suit particular needs. 

Insight into family historians preferred mode of research, contributed to a slightly different 1st and 2nd tier 
"merging". This group relishes the common purpose they share with their colleagues. They love to know 
what their colleagues are doing and what they can learn from them. Many told us they discovered new 
resources or ways of doing things by talking to other researchers. Taking this into account, we are just 
about to trial a new service that will include a brief instruction session that will cover new family history 
resources, or "hands on" technology. Run by staff, the session will continue as a physical "chat room" that 
will tap Into the sometimes club like and serendipitous way family historians work. This allows family 
historians to draw both on the skills of each other and staff. We are also investigating the possible 
development of a virtual chat room for family historians. 

Other value outcomes relate to reducing complexity and increasing client independence. We recognise the 
complexities faced by first time users when they visit the State Library. Many struggle to understand and 
make sense of what they are experiencing but are reluctant to approach the Information Desks to ask for 
help. They are concerned about sounding like idiots, or wasting the staffs time so try to figure it out for 
themselves. 

A major element of the 1st Tier is the promotion of client independence and self sufficiency. Accordingly, our 
1st Tier incorporates a range of products and services from roving staff to electronic and paper based 
finding aids such as FAQs, self guided tours, introductory tours and "virtual" tours. The value for these 
clients is in having the guesswork taken out of their introduction to the Library. 

We find that a high proportion of questions are simple directional or repetitive questions. (Where are the 
lockers? Can I join this Library?) The first information source visitors approach are the foyer desks. For 
many years, senior Library staff worked on one desk and Volunteers on the other. This resulted in patchy 
service. We realised that it was critical that the responsibility at this desk be well defined and that only 
directional and general questions be answered. So we introduced a new group of staff known as Information 
Officers, whose prime purpose was to provide a "front of Library" service that introduced people to the 
Library and its’ facilities. This in turn freed the professional staff to carry out more specialist work and the 
volunteers to do move In to other support activities 



Staff adding value 



Personal experience affects how clients perceive the service value of staff. The attitudes of staff, particularly 
if they value the client's time and deliver what they asked for, are integral to a client's positive experience in 
the Library. 

What attributes ensure effective information professionals? Providing accurate, relevant information is a 
complex process that requires a blending of information finding skills, interpersonal skills and cognitive 
processes. An ability to conceptualise information is promoted by some to be the core competency (as 
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opposed to the core skills) that librarians need.2 This means posing the right questions and retrieving only 
relevant information for the client. It also indicates filtering and evaluating content and saving the client 
valuable time. Core skills are also important but these change over time. Computer skills are an example of 
this. 

The increasing presence of technology does not eliminate the need for human intervention - rather, in some 
cases, it has heightened it. Staff are a critical component in helping the client access and navigate an often 
bewildering information environment. The Health Information Service, for example, has developed a tool to 
help clients assess the quality of health information on the 'Net. This is in the form of a checklist that covers 
authority, content, design and relevance. Health staff also utilise other skilled professionals who have 
specific subject skills (such as the Cancer Council) to assess health sites. 

In the 3Tler Model it is not sufficient to focus solely on service outcomes. We need to strongly underpin this 
service model with adequate support, training, resources and tools. The new model is based on staff having 
a range of core skills and competencies. Position Descriptions now include elements such as "analytical 
skills, client focus, communication and interpersonal skills". 

William Bridges has made an important point and one that is pivotal to us moving our services forward at 
the State Library. He draws a distinction between change that is situational (e.g. new team roles) ^nd 
transitional change which is the psychological process we need to go through to come to terms with new 
situations. At the State Library, we need to balance traditional activities such as knowledge of information 
and library collections and meld these with new service initiatives and additional skills. The 3Tier Model 
requires an explicit client oriented approach to services - one in which we need to realign and reshape 
services and resources in response to the changing world of information and the changing needs of our 
clients. 

Collecting the data 

Client research and the development of new services highlights the difficulty in establishing consistent 
measures of the Library's value and relevance. Our experience suggests that quantifying these concepts in a 
meaningful way can be problematic. Rather, it is a matter of recognising these sometimes 'irrational' 
attitudes that very often relate to external issues, and the effect they have on clients' service perceptions at 
the Library. 

This being the rase, we have developed a Library wide measurement model that takes client satisfaction as 
its central measure. This model is consistent, yet also flexible enough to incorporate the diversity of Library 
services. 

We define client satisfaction as a composite measure, underpinned by a product, service and if applicable, 
price factor. We also allow for the 'x' factor, the usually intangible, often 'irrational' element that affects a 
client's perception of satisfaction. This factor is a broader value judgment or perception that a client brings 
to the Library, for example a reader's perception that the Mitchell Library is valuable due to its collection. 
The following mode! (Figure one) outlines the standard relationship between the variables. 
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Figure one 

Our research has shown that client satisfaction is a reliable predictor of client behaviour: if a client’s 
satisfaction is high, they are most likely to come again and tell others about the service. 

Our focus, when considering a product and/or service is the quality of that product or service. We consider 
that rating the quality is a far more critical and objective measure than asking a client questions such as 
'Did you find the information you require?' While it is important to know such information, questions like this 
are not a reliable indicator of client satisfaction as they do to not take into consideration the various factors 
(such as the client's research skills) that can influence the end result. This is why, from a provider 
perspective it is best to focus measurement on the service we provide. 

In some instances we ask clients to consider the overall quality of a service and/or product. However, in 
order to gain a more meaningful understanding of what a client values and finds relevant regarding a 
particular service, it is often essential to understand what, for a client, defines the service. The important 
point is that defining criteria differ from service to service. Without uncovering the attributes for individual 
services, we run the risk of measuring the wrong attribute, which in turn leads to incorrect conclusions as to , 
a service's performance. A case in point is the measurement of service provided by reading room staff. In 
the Mitchell Library, the key attribute for readers is that staff provide a consistent service. However, in the 
SRL, the most important attribute is the friendliness of the staff. To have only measured one of these 
attributes across both reading rooms would have meant that we were not providing a valid indicator of 
quality service. 

Most importantly, this performance model enables us to identify the priority areas; aspects of a service that 
have the highest impact on a client's satisfaction rating, yet rank the lowest levels in performance. 
Highlighting these areas ensures that we dedicate limited resources to the most appropriate areas. Table 
one highlights the priority areas for the Mitchell Library. 
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Quality service at the Mitchell Library 



Key service attributes and top ten factors 



Access to the 
libratY 


Guidance & 
instructions 


Staff 


Requesting 

material 


Library 

environment 


Technology 




Instructions for 
computer 




Notification 
material is ready 
to collect 




Costing of 
copying 




Instructions for 
Indexes ■ 




Notification of 
request's 
progress 








Guides & 
brochures 




Help with material 
unavailable 








Signage 




Waiting time 












Number of items 
able to request 












Stack slips 







Table one 



Benchmarking 

Our approach to performance definition and measurement means that it will be a while before the Library 
has comprehensive performance indicators for all of the Library's services. It also indicates that it will be 
sometime before we are ready to compare ourselves with competitors from other industries. 

This is not to suggest that we are not 'benchmarking' our services. In fact we are benchmarking In ways we 
consider are more appropriate to our current context. 

One major component to our internal benchmarking process is measuring a service before changes are 
introduced and then, once those changes have been implemented and given time to establish, re-measuring 
the key attributes. The focus of this measurement and benchmarking are specific attributes that define the 
service, for example, when measuring the performance of the Library's Picture Search Service, the key 
attributes Included the Information on the service, the request process and payment. Within these 
attributes, specific factors were measured, for example, the main criteria for payment was that invoices 
carried enough information for the client. Knowing which attributes to include and the specific factors 
defining those variables is uncovered by conducting qualitative research, in this case, in-depth interviews 
with a selection of clients. 

This benchmark activity provides us with not only an objective indicator of the success of new service 
developments, but very specific Information as to the performance of the various aspects of the service. The 
three tier service model Is an example of where we have sought client feedback before the implementation 
of the service, and where we will ensure client perceptions of performance are measured, once the service is 
operational. 

Another benchmark exercise we practice is comparing key quality measures across services. Comparison 
enables us to gauge what, from a client perspective, are the Library's quality services. This process also 
helps us Identify 'best practice' services and highlights the attributes that make those services so 
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exceptional. 

We also Identify best practice competitors by asking clients to tell us who they think provides the highest 
quality service or product within a particular area. At a later stage, we will be able to use the information in 
external benchmarking exercises. 

At this point in time, we consider all of the above exercises to be of more benefit to us than comparing 
ourselves to external competitors from another industry. However, once we have established the services 
we need to provide to clients and have then developed an appropriate number and balanced set of 
benchmarks, we will consider external benchmarking as a way of ensuring that we reach and maintain the 
highest level of performance that can be possibly achieved. 



Footnotes 

1 Nunberg, G. (1998) "Will libraries survive?". The American Prospect online, Issue 41 November- 
December, http://www.prospect.org 

2 Nichols, M., Sikes, J., Isselman, M., Ayers, M., Seelig, R. (1996) "Survival in transition or implementing 
information science core competencies". Bulletin for the American Society Tor Infonmation Science, American 
Society for Information Science, Dec - Jan issue 

3 Bridges, W. (1991) Managing Transitions: Making the Most of Change, Reading, Mass., Addison-Wesley 
Publishing Co. 
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